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We investigate a limit value of an optimal control problem when the horizon converges 
to infinity. For this aim, we suppose suitable nonexpansive-like assumptions which does 
■ not imply that the limit is independent of the initial state as it is usually done in the 

^ . literature. 



Abstract 



1 Introduction 



1 /•* 

(1) Vtivo) := mf - h{y{s,u,yo),u{s))ds 



^ ■ We consider the following optimal control denoted Tt{yo) '■ 

m 
vn 

■ ""^"^ Js=0 

g; (2) y'{s)=g{y{s),u{s)), y{0) = yo- 

Here U is the set of measurable controls from to a given non empty metric 
^ I space U. Throughout the paper, we will suppose Lipschitz regularity of g : M'^ x 

^1 U ^ IR^ which implies that for a given control u'mlA and a given initial condition 

?/o, equation ([21) has a unique absolutely continuous solution. 

The main goal of the paper consists in studying the asymptotic behaviour of 
Vt{yo) when t tends to oo. This problem has been considered in several papers 
(cf for instance in [21 [TJ E]) by approaches ensuring that the limit of Vtiyo) is 
independent of y^. In the present paper we exhibit several examples where the 
limit exists and depends of y^. Our aim is to obtain a general result which contains 
in particular the more easy to state following result, where throughout the paper, 
< ■, ■ > stands for the canonical scalar product and B is the associated closed 
unit ball.. 
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Proposition 1.1. Assume that g is Lipschitz, that there exists a compact set N 
which is - forward - invariant by the control system ^ and that h is a continuous 
function which does not depend on u. Assume moreover that : 

(3) V(yi, 7/2) e sup inf <yi- y2, g{yi, u) - g{y2, v) >< 0. 

Then problem ^ has a value whent converges to +00 i.e. there exists V^(?/o) '■= 
liiiif^+oo Vt(?/o)- 

Condition ([3]) means a non expansive property of the control system, while 
the condition 



expresses a dissipativity property of the control system. The above dissipativity 
condition does imply that the limit is independent of y^ (cf [3]). 

The value function ([1]) can also be characterized through - viscosity - solution 
of a suitable Hamilton- Jacobi equation. In several articles initiated by the pio- 
neering work [T2j the limit of V^(|/o) is obtained by "passing to the limit" on the 
Hamilton- Jacobi equation. This required coercivity properties of the Hamiltonian 
which could be implied by controlability and/or dissipativity of the control sys- 
tem but which are not valid in the nonexpansive case ([3]). Moreover the PDE 
approach is out of the scope of the - long enough - present article. 

Definition 1.2. The problemV [y^) := (rt(?/o))t>o has a limit value z/limj^oo Vt(?/o) 
exists. Whenever it exists, we denote this limit by Viyo). 

Our main aim consists in giving one sufficient condition ensuring the existence 
of the limit value. As a particular case of our main result we obtain proposition 



It is also of interest to know if approximate optimal controls for the value 
Vtiyo) are still approximate optimal controls for the limit value. This leads us to 
the following definition. 

Definition 1.3. The problem r(?/o) has a uniform value if it has a limit value 
V{yQ) and if : 



Whenever the uniform value exists, the controller can act (approximately) 
optimally independently of the time horizon. On the contrary, if the limit value 
exists but the uniform value does not, he really needs to know the time horizon 
before choosing a control. We will prove that our results do imply the existence 
of a uniform value. We will be inspired by a recent work in the discrete time case 



Let us explain now, how the paper is organized. The second section contains 
some preliminaries and discussions of limit behaviors in examples. In the third 
section, we state and prove our main result for the existence of the uniform value. 



V(i/i, y2) e sup inf < yi - y2, g{yi, u) - g{y2, v) >< -C\\yi - y2 
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2 Preliminaries 

We now consider the optimal control problems {Tt{yo))t described by ([1]) and 

2.1 Assumptions and Notations 

We now describe the assumptions made on g and h. 



(4) 



The function h : M'^ x U — > M is measurable and bounded 
The function g : M'^ x U — > M'^ is measurable 
3L > 0, V(y, y') G iR2rf, G U, \\g{y, u) - g{y\ u) \\ <L\\y-y'\ 
3a > 0,V(y,n) e R'' x U, \\g{y,u)\\ < a{l + \\y\\) 



With these hypotheses, given u in U equation ([2]) has a unique absolutely 
continuous solution y{-,u,yo) : M+ M'^. 

Since h is bounded, we will assume without loss of generality from now on 
that h takes values in [0, 1]. 

We denote by G{yo) := {y(t,u,yo),t > 0,m G U} the reachable set (i.e. the 
set of states that can be reached starting from yo). 

We denote the average cost induced by u between time and time t by : 
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lt{.ya,u) = - j h{y{s,u,yo),u{s))ds 
^ Jo 

The corresponding Value function satisfies Vt{yo) = inf^g^^ 74(7/0, «)• 



2.2 Examples 

We present here basic examples. In all these examples, the cost h{y, u) only 
depends on the state y. We will prove later that the uniform value exists in 
examples 2, 3 and 4. 

• Example 1 : here y lies in seen as the complex plane, there is no control 
and the dynamic is given by giy, u) = i y, where i"^ = —1. We clearly have : 

VtiVo) ^ lT~r~\ I h{z)dz, 

and since there is no control, the value is uniform. 

• Example 2 : in the complex plane again, but now g{y,u) = i y u, where 
u E U a. given bounded subset of M, and h is continuous in y. 

• Example 3 : g{y, u) = —y + u, where u E U a given bounded subset of M"^, 
and h is continuous in y. 
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• Example 4 : in M^. The initial state is = (0, 0) and the control set 
is U = [0, 1]. For a state y = (1/1,1/2) and a control u, the dynamic is given by 

y'{s) = g{y{s),u{s)) = ^i^^j^^^ -^yi{s)) ) ' ^^^^ ^ l-yi(l-|/2). 

Notice that for any control, y'i{s) > y2{s) > 0, and thus y2{t) < yi{t) for each 
t > 0. One can easily observe that G{yo) C [0, 1]^. 

If one uses the constant control u = e > 0, v/e obtain yi{t) = 1 — exp{—et) 
and y2{t) — eyi{t). So we have Vt{yo) 0. 

t—*oo 

More generally, if the initial state is y = (yi,?/2) G [0)1]^) by choosing a 
constant control u — e > small, one can show that the limit value exists and 
liuit^^Vtiy) = 1/2 • 

Notice that there is no hope here to use an ergodic property, because 

{y e [0, 1]^ hm Vt{y) = lim VM} = [0, 1] x {0}, 

t^OO E— >00 

and starting from yo it is possible to reach no point in (0, 1] x {0}. 

• Example 5 : in M^, yo = (0,0), control set U = [0, 1], g{y,u) = (1/2, w), and 
Hvi^ ?/2) = if yi e [1, 2], = 1 otherwise. 

We have u{s) — 1/2 (*) — Vii^)-! hence we may think of the control u as the 
acceleration, 1/2 as the speed and yi as the position of some mobile. \i u — £ 
constant, then 1/2 (t) = ^J2eyl{t) Vt > 0. 

We have m > 0, hence the speed cannot decrease. Consequently, the time 
interval where yi{t) G [1,2] cannot be longer than the time interval where y\{t) G 
[0, 1), and we have Vt(?/o) > 1/2 for each T. 

One can prove that VT(yo) ^ 1/2 by considering the following controls : 

T— »oo 

choose t in (0,T) such that (2/t) + (t/2) = T, make a full acceleration up to t 
and completely stop accelerating after : u{t) = 1 for t < t, and u{t) = for t >t. 

Consequently the limit value exists and is 1/2. However, for any control u in 
U, we either have y{t, u, yo) = yo for all t, or yi(t, u, yo) > +00. So in any case 

t— >oo 

we have j h{y{s, u, yo) , u{s))ds > 1. The uniform value does not exist here, 

although the dynamic is very regular. 

3 Existence results for the uniform value 
3.1 A technical Lemma 

Let us define V~{yo) := liminff_»+oc Vt{yo) and V+{yo) := limsup^^+oo Vtiuo)- 
Adding a parameter m > 0, we will more generally consider the costs between 
time m and time m + t: 

7m,t(yo,ii) = T / h{y{s,u,yo),u{s))ds, 
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mo 



and the valnc of the problem where the time interval [0, m] can be devoted to 
reach a good initial state, is denoted by : 

Vm,t{yo) = inf 7m,t(Z/0,1i)- 

Of course 7t(|/o,M) = 7o,t(|/o,M) and Vt{yo) = Vo,t{yo)- 
Lemma 3.1. For every mo in M+, we have : 

sup inf Vm,tiyo) > V^{yo) > V~{yo) > sup inf VmAvo)- 

t>0 m<mo t>0 "^>0 

Proof : We first prove sup^^o infm<mo Kn,t(?/o) > V^{yo). Suppose by contra- 
diction that it is false. So there exists e > such that for any t > we have 
mim<mo Kn,t(l/o) < ^^(l/o) ~ ^ • Hcuce for any t > there exists m < mo with 
Vm,t{yo) < V^iVo) - (£/2)- Now observe that 

Vm,t{yo) = inf - / h{y{s, u, yo),u{s))ds = - inf{ / h{y{s, u, yo),u{s))ds 
-/ h{y{s,u,yo),u{s))ds - h{y{s,u,yo),u{s))ds} > ^ — Vmo+tivo) - 2 

Jm+t Jo t 

Hence 

^V^Myo)-2^<V^yo)-{e/2). 

Passing to the limsup when t goes to +oo we obtain a contradiction. 

We now prove V^{yo) > sup^^g infm<o Kn,t(l/o)- Assume on the contrary that it 
is false. Then there exists £ > and t > such that V~{yo)+e < inf^<o Vm,t{yo)- 

So for any m > 0. we have V~{yo) + e < V„i.t{yo)- We will obtain a contradiction 
by concatenating trajectories. Take T > 0, and write T = It + r, with / in IV and 
r in [0, t). For any control u in U, we have : T7t(i/o, u) = ^7o,t(yo, u) + t'yt,t{yo, u) 
+ ... + t'y(i-i)t,tiyo,u) + r'jit,riyo,u) > lt{V-{yo) +e). Hence 

T -r 

Myo,u) > {yo) + e). 

So for T large enough we have Vriyo) > ^"(z/o) + hence a contradiction by 
taking the liminf when T — > oo . □ 

Remark : it is also easy to show that for each to > 0, we have inf^>o supj^^^ Vmt{yo) > 
V^yo). 

The following quantity will play a great role in the sequel. 
Definition 3.2. 

V*{yo) =sup inf Vm,t{yo)- 
t>o 
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3.2 Main results 



Let us state the first version of our main result (wliicli clearly implies Propo- 
sition 11.11 stated in the introduction) 

Proposition 3.3. Assume that ^ holds true and furthermore : 

(H'l) h{y,u) = h{y) only depends on the state, and is continuous on IR^. 

(H'2) Giyo) is bounded, 

(H'3) V(?/i, 1/2) e G{yoy, snp^^u inf^g^/ < yi - y2, g{yi, u) - g{y2, v) >< 0. 
Then the problem r(?/o) has a limit value which is V*{yQ), i.e. Vt{yo) > 

V*{yQ). The convergence of {Vt)t to V* is uniform over G{yo) , and we have 
V*{yo) = supi>i infm>o K^,^(2/o) = inf„>o supj>i Kn,t(2/o) = lim^^oo.t^oo Ki,t(2/o)- 
Moreover the value o/r(?/o) is uniform. 

Condition {H'3) can be used to show that (cf Proposition 13.51) : V(?/i,y2) ^ 
G(yoy , Ve > 0, VT > 0, \/u e U, 3v E U s.t. : Vt G [0,T], \\y{t,u,yi) - 
y{t,v,y2)\\ < \\yi — 1/2II + £■ Proposition 13.31 can be applied to the previous 
examples 1, 2 and 3, but not to example 4. Notice that in example 5, we have 
V*iyo) = 0<l/2 = limtVtiyo). 

We will prove the following generalization of Proposition 13.31 We put Z = 
G{yo), and denote by Z its closure in M'^. 

Theorem 3.4. Suppose that holds true and furthermore assume that 
(HI) h is uniformly continuous in y on Z uniformly in u. And for each y in Z , 
either h does not depend on u or the set {{g{y, u), h{y, uj) G M'^ x [0, 1], u E U} 
is closed. 

(H2) : There exist a continuous function A : x IR^ — > ]Rj^, vanishing on the 
diagonal (A{y,y) = for each y) and symmetric (A{yi,y2) = A{y2,yi) for all yi 
and y2), and a function a : iR+ — > iR+ s.t. a{t) > satisfying : 

a) For every sequence {zn)n with values in Z and every e > 0, one can find n 
such that liminfp A(z„, Zp) < e. 

b) V(2/i,y2) e Z^, \fu eU, 3v eU such that 

D T ^{yuy2)igiyi,u), g{y2,v)) < O and h{y2,v) - h{yi,u) < a{A{yi,y2)). 

Then we have the same conclusions as in Proposition \3.3[ The problem r(?/o) 
has a limit value which is V*{yo). The convergence of Vt to V* is uniform over 
Z, and we have l^*(?/o) = ^'^Vt>\ infm>o Kn,t(2/o) = vaim>s)^\yY>t>\ VmAvo) = 
Hmm^oo,t^ooVm,t{yo) ■ Morcovcr thc value ofT^yo) is uniform. 

Remarks : 

• Although A may not satisfy the triangular inequality nor the separation 
property, it may be seen as a "distance" adapted to the problem TIjjq). 

• The assumption : "{((?(?/,«), h{y,u)) G iR'' x [0, 1], u E U} closed" could be 
checked for instance if U is compact and if h and g are continuous with respect 
to {y,u). 
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• D I is the contingent epi-derivative (cf [6J) (which reduces to the upper Dini 
derivative if A is Lipschitz), defined by : D^A{z){a) = hminft^o+,o'^o li^i^ + 
ta') —A{z)). If A is different iable, the condition D t A(?/i, y2){g{yi, u), g{y2, v)) < 
just reads : < g{yi, u), g^A(?/i, 7/2) > + < g{y2, v), g^A(yi, 1/2) >< 0. 

• Proposition 13.31 will be a corollary of Theorem |3.4[ It corresponds to the 
case where : A{yi,y2) = \\yi — y2\f, G{yo) is bounded, and h{y,u) = h{y) does 
not depend on u (one can just take a{t) = sup{\h{x) — h{y)\, \\x — y\f < t}). 

• H2a) is a precompacity condition. It is satisfied as soon as G(|/o) is bounded. 
It is also satisfied if A satisfies the triangular inequality and the usual precom- 
pacity condition : for each e > 0, there exists a finite subset C of Z s.t. : 
Vz e Z, 3c e C, A{z, c) < e. (see lemma Km 

• Notice that H2 is satisfied with A = if we are in the trivial case where 
inf uh{y,u) is constant. 

• Theorem 13.41 can be applied to example 4, with A(?/i,?/2) = \\yi — I/2II1 (L^- 
norm). In this example, we have for each yi, y2 and u : A{yi + tg{yi,u),y2 + 
tg{y2,u)) < A(?/i,?/2) as soon as t > is small enough. 



3.3 Proof of Theorem [374] 

We assume in this section that the hypotheses of Theorem 13.41 are satisfied, 
and we may assume without loss of generality that a is non decreasing and upper 
semicontinuous (otherwise we replace a{t) by infe>o supj/g^^^^^j a(t')). 



3.3.1 A non expansion property 

We start with a proposition expressing the fact that the problem is non ex- 
pansive with respect to A, the idea being that given two initial conditions yi and 
?/2 and a control to be played at yi, there exists another control to be played at 
?/2 such that 1 1— > A{y(t, u, yi) , y(t, V, y2)) will not increase. 



Proposition 3.5. We suppose the hypothesis of Theorem \3.4\ Then 
(5) 



V(i/i,i/2) e Z^, VT > 0, V£ > 0,Wu eU,3ve U, 
Vt G [0,T], Aiyit,u,yi),yit,v,y2)) < A{yi,y2)+e, 
and for almost every t G [0, T], 
h{y{t,v,y2),v{t)) - h{y{t,u,yi),u{t)) < a{A{y{t,u,yi),y{t,v,y2))). 



Proof : First fix yi,y2 e > 0,T > and u. Let us consider the following set- valued 
map ^ : R+ X Z X Z X R'^ X R'^ X IR 

^{t,x,yj) := cocl{{g{x,u{t)), g{y,v),0)) \ v e U, h{y,v)-h{x,u{t)) < a{A{x,y))}, 

where co stands for the convex hull and cl for the closure. Notice that ^{t, x, y, I) 
does not depend on 1. Using HI) and H2)b), one can check that $ is a set 
valued map which is upper semicontinuous in {x,y,l), measurable in t and with 
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compact convex nonempty values P, ITU]. We also denote $ the set valued map 
defined as $ but removing the convex hull. 

^From the measurable Viability Theorem [11] (cf also [9] section 6.5), condi- 
tion (H2) b) implies that the epigraph of A (restricted to x M) is viable for 
the differential inclusion 

(6) (x (t), y'(t), /'(t)) e ^(t, x{t),y{t), l{t)) for a. e. t > 

So starting from 1/2, A(?/i, ?/2)), there exists a solution (a;(-), ?/(■), /(■)) to ([6]) 
which stays for any t > in the epigraph of A namely 

(7) A{x{t),y{t)) < l{t) = A(yi,y2), Vt > 0, 

by noticing that /(■) is a constant. 

/^From the suppositions made on the dynamics g, the trajectory (a;(-),y(-)) 
remains in a compact set (included in some large enough ball 5(0, M)) on the 
time interval [0,T]. Because A is uniformly continuous on 5(0, M) x 5(0, M), 
there exists 77 G (0, 1) with 

W{x,x',y,y') e 5(0, M + l)"^, + < r] ^ \A{x,y) - A{x' ,y')\ < e. 

Thanks to the Wazewski Relaxation Theorem (cf for instance Th. 10.4.4 in 
[6]) applied to $, the trajectory (x(-), ?/(■), /(■)) could be approximated on every 
compact interval by a trajectory to the differential inclusion defined by $. So 
there exists {yi{-),y2{,-),l{')) satisfying 

{y[{t),y'^{t),l'{t)) e ^t,yi{t),y2{t),lit)) for a. e. t > 

such that 

\\x{t)-y,{t)\\ + \\y{t)-y2m<V, VtG [0,r]. 
From the choice of 1] and the very definition of $ we also have for any t G [0, T] 

r A(t/i(t),y2(t)) < A{x{t),y{t))+e< A{y^,y2)+e 
\ h{y2{t),v{t)) - h{y^{t),u{t)) < «(A(yi(t),?/2(t))) 

This completes our proof if, from one hand we observe that yi{-) = y{-,u,yi) 
and from the other hand, we use Filippov's measurable selection Theorem (e.g. 
Theorem 8.2.10 in [6]) to $ for finding a measurable control w G W such that 
y2{-) = y{-,v,y2). 

QED 

3.3.2 The limit value exists 

Since a is u.s.c. and non decreasing, we obtain the following consequence of 
Proposition 13. 5[ 
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Corollary 3.6. For every yi and y2 in G{yo), for all T > 0, 

\VTiyi)-VTiy2)\ < d(A(yi,|/2)). 

Define now, for each m > 0, G"*(2/o) as the set of states which can be reached 
from Xq before time m : 

G""(yo) = {yit, u,yo),t <m,u e U}, so that G{yo) = U„>oG""(?/o). 

An immediate consequence of the precompacity hypothesis H2a) is the following 

Lemma 3.7. For every e > 0, there exists mo in IR+ such that : 

G G{yo), 3z' G G""o(?/o) such that A(2, z') < e. 

Proof : Otherwise for each positive integer m one can find Zm in G{yo) such that 
A(2;m, z) > e for all z in G"^{yo). Use H2a) to find n such that liminfm A(z„, Zm) < 
£. Since z„ G G{yo), there must exist /c such that z„ G G^iyo). But for each m>k 
we have 2;„ G G"^{yo), hence A{zm, Zn) > £■ We obtain a contradiction. 

QED 

We can already conclude for the limit value. 
Proposition 3.8. Vt{yo) > V*{yo). 

t— >cx> 

Proof : Because of lemma [3TT1 it is sufficient to prove that for every e > 0, there 
exists mo such that : 

sup inf Vm,t{yo) < sup inf Vm,tiyo) + 

Fix e, and consider rj > such that a{t) < £ as soon as t < r/. Use lemma 13771 to 
find mo such that G G(?/o), 3z' G G'^od/o) s.t. A(2, z') < r]. 

Consider any t > 0. We have infm>o Kn,t(l/o) = inf{Vt(z),2; G Gd/o)}, and 
inf„<mo Vm,t{yo) = mi{Vt{z),z G G'"'o(?/o)}. Let z in G(?/o) be such that Vt{z) < 
inirnVm^tiyo) + and consider z' G G™"(?/o) s.t. A(z, z') < 77. By corollary 13. 6[ 
\Vt{z) — Vt{z') \ < a{A{z,z')) < e, so we obtain that 

inf Kn,t(yo) < Vt{z') < Vt{z) + e < infV^^tiyo) + 2e. 

m<mo rn 

Passing to the supremum on t, this completes the proof. 

QED 

Remark 3.9. Observe that for obtaining the existence of the value, we have used 
a compactness argument (assumption H2)a)) and condition We did not use 
explicitly assumption H2)b) which is only used for obtaining Ij^. 

The rest of the proof is more involved, and is inspired by the proof of Theorem 
3.6 in [I3j. 
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3.3.3 Auxiliary value functions 

The uniform value requires the same control to be good for all time horizons, 
and we are led to introduce new auxiliary value functions. Given m > and 
n > 1, for any initial state z in Z = G{yo) and control u in U, we define 

^m,n{z,u) = sup 7„.t(z,u), ciiad Wm.n{z) = iuf Urn,n{^^u). 
te[l,n] ""6" 

Wm,n is the value function of the problem where the controller can use the time 
interval [0, m] to reach a good state, and then his cost is only the supremum for 
t in [l,n], of the average cost between time m and m + t. Of course, we have 
Wm,n > Vm,n- We Write Vn for z/o,„, and Wn for Wo,n- 

We easily obtain from proposition 13. 5[ as in corollary 13. 61 the following result. 

Lemma 3.10. For every z and z' in Z , for all m > and n > 1, 

\Vm,n{^) -Vm,n{z')\ < a{A{z,z')). 
\W^,n{z) -Wm,nW)\ < a{A{z,z')). 

The following lemma shows that the quantities Wm,n are not that high. 
Lemma 3.11. Vfc > l,Vn > l,Vm > 0,V2 G Z, 

k 

Vm,n{z) > inf Wi^kiz) - -. 

l>m n 

Proof : Fix k, n, m and z, and put A = inf/>m Wi^k{z)- Consider any control u 
in U. For any i > m, we have 

sup ji^t{z,u) = Ui,k{z,u) > Wi^k{z) > A. 
te[i,k] 

So we know that for any i > m, there exists t{i) G [1, k] such that 7j u) > A. 

Define now by induction ii = m, i2 = ii + t{ii),..., ig = + where q 

is such that ig < n + m < iq + t{iq). We have n'-frn,n{z,u) > ^21=1^^)^ — nA — k, 
so 7m,n(-2, u) > A — K Taking the infimum over all controls, the proof is complete. 

QED 

We know from Proposition 13.81 that the limit value is given by V*. We now 
give other formulas for this limit. 

Proposition 3.12. For every state z in Z , 

inf sup Wra,n{z) = iuf SUp Vm,n{z) = V* {z) = SUp iuf Vm,n{z) = SUp luf Wra,n{z) ■ 
m>0 n>l m>0 „>i „>i m>0 „>i m>0 
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Proof of proposition 13.121 : Fix an initial state z in Z. We already have 
V*{z) = supi>o infm>o Vm,t{z) > sup4>i infm>o Vm,t{z). One can easily check 
that infm>o Vm,t{z) < infm>o Vm,2t{z) for each positive t. So 

V*{z) > sup inf V^^t{z) > sup inf Vm,t{z) > . . . sup inf Vm,t{z) = V*{z). 

t>l '»>0 t>{l/2) i>0 ''»>0 

Consequently V*{z) = sup^i infm>o Vm,t{,z). Moreover because Vm,t < W^m,t we 
have also V*{z) < sup^^i infm>o Wm,t{z). 

We now claim that V*{z) = sup^i infm>o Wm,t{z). It remains to show 
V*{z) > snpf^i infm>o Wm,t{z)- From Lemma [3.111 we know that for all A; > 1, 
n > 1 and m > 0, we have Vm,nk{z) > mfi>oWi^k{z) - ^, so inf^ V;„^„fc(2;) > 
inf;>o Wi^k{z) — ^- By taking the supremum on n , we obtain 

V*{z) = sup inf Vm,n{z) > sup inf Vm,nk{z) > miWi^kiz). 

n>l "i>0 n>l 'Ti>0 l>0 

Since k is arbitrary, we have proved our claim. 
Since the inequalities 

inf sup Wm,n{z) > inf sup Vm,n{z) > sup inf Vm,niz) = V*{z) 

m>0 „>i m>0 „>i „>i m>0 

are clear, to conclude the proof of the proposition it is enough to show that 
infm>osup„>i Wm,n{z) < V*{z). 

Fix e > 0. We have already proved that V*{z) = sup„>i infm>o Wm,n{z), so for 
each n> 1 there exists m > such that Wm,n{z) < y*{z) + £■ Hence for each n, 
there exists z'^ in G{z) such that WQ^n{z'^) < V*{z) + e. We know from Lemma [3171 
that there exists mo > such that : \/z' E G{z), 3z" e s.t. A{z' , z") < e. 

Consequently, for each n > 1, there exists z'^ in G^'^^z) such that /S.{z[^,z'^) < e, 
and by lemma 13.101 this implies that 

Wn{z';) < Wniz'J + a{e) <V*{z)+e + a{e). 

Up to now, we have proved that for every e' > 0, there exists tuq such that : 

Vn > 1, 3m < mo s.t. Wm,niz) < V*{z) + e'. 

Since all costs lie in [0, 1], it is easy to check that |W^m,n(^) — Wm',n{z)\ < |m — m'| 
for each n, m, m'. Hence there exists a finite subset F of [0,mo] such that : 
Vn > 1, 3m G F s.t. Wm,n{z) < V*{z) + 2e'. Considering m in F such that the set 
{n positive integer, Wrh,n{z) < V*{z) + 2e'} is infinite, and noticing that Wm,n is 
non decreasing in n, we obtain the existence of a unique m > such that Vn > 
1, Wrh,n{z) < V*{z) + 2e' . Heuce e' being arbitrary, infm>oSup„>i Wm,n{z) < 
V*{z), concluding the proof of Proposition I3.12[ 

QED 
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We now look for uniform convergence properties. By the precompacity condi- 
tion H2a), it is easy to obtain that : 

Lemma 3.13. For each e > 0, there exists a finite subset C of Z s.t. : G 
Z,3ceC,A{z,c) < e. 

We know that (V"„)„ simply converges to V* on Z. Since — — 

d;(A(z, z')) for all n, z and z' , we obtain by lemma 13.131 : 

Corollary 3.14. The convergence of {Vn)n to V* is uniform on Z. 

We can proceed similarly to obtain other uniform properties. We have 
V*{z) = sup inf Wm,n{z) = lim inf Wm,n{z) 

n>l ra^+oo m>0 

since mim>oWm,n{z) is not decreasing in n. Using lemmas 13.101 and I3.13[ we 
obtain that the convergence is uniform, hence we get : 

Ve > 0, 3^0, Vz G Z, V*{z) -e< inf W„r,no{z) < V*{z). 
By Lemma I3.1H we obtain : 

Ve > 0,3no,V2 G Z,Vm > 0,Vn > l,K^,„(z) > V*{z) -e-—. 

Th 

Considering n large gives : 

(8) Ve > 0, 3K, yz eZ,yn> K, inf Kn,„(2) > V*{z) - e 

m>0 ' 

Write now, for each state z and tti > : hm{z) — infm'<m sup^>]^ 
{hm)m converges to V*, and as before, by Lemmas 13. 101 and 13. 13[ we obtain that 
the convergence is uniform. Consequently, 

(9) ye > 0, 3M > 0, eZ,3m< M, sup Wm,n{^) < V*{z) + e. 

n>l 

3.3.4 On the existence of a uniform value 

In order to prove that T{yo) has a uniform value we have to show that for 
every e > 0, there exist a control u and a time no such that for every n > no, 
'jnivo, u) < V*{yo) + e. In this subsection we adapt the proofs of Lemma 4.1 and 
Proposition 4.2. in [T3|. We start by constructing, for each n, a control which : 

1) gives low average costs if one stops the play at any large time before n, and 

2) after time n, leaves the player with a good "target" cost. This explains the 
importance of the quantities i'm,n- We start with the following 
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Lemma 3.15. > 0, 3M > 0, 3K > 1, Vz G Z, 3m < M,\/n > K,3u such 
that : 

(10) Urn,n{z,u)<V*{z)+e/2, Slid V* {y{m + u, z)) < V* {z) + E . 

Proof: Fix £ > 0. Take M given by ([9]), so that G 3m < M, sup„>i Vrm,„(2) < 
V*{z) + e. Take > 1 given by ® such that : G Vn > fsT, inf^ Kn,„(2) > 
V*\z)-e. 

Fix an initial state z in Z. Consider m given by ([9]), and n > K. We have to 
find M in W satisfying (fTOl) . 

We have Wm,n'{z) < V*{z) + £ for every n' > 1, so Wm,2n{z) < V*{z) + e, 
and we consider a control u which is e-optimal for Wm,2n{z), in the sense that 
^m,2n{z, u) < Wm,2n{z) + s. We have : 

Um,n{^^ m) < Vni,2n{z, u) < Wm,2n{z) + E <V*{z) + 2e. 
Denoting X = ^ra,n{z, U) and Y = Jm+n,n{^^ 

X Y 

m m + n m + 2n 

Since z/^,2n(^,M) < V*{z) + 2e, we have X < V*{z) + 2e, and (X + Y)/2 = 
lm,2n{.z,u) < V*{z)+2e. Sincc n > K, we also have X > Vm,n{z) > V*{z)—e. And 
n > K also gives Vn{y{m+n, u, z)) > V*{y{m+n, u, z))—e, so V*{y{m+n, u, z)) < 
Vn{y{m + n,u,z)) + e <Y + e. Writing now Y/2 = {X + Y)/2- X/2 we obtain 
Y/2 < {V*{z) + 56)/2. SoY< V*{z) + 5e, and finally V*{y{m + n,u,z)) < 
V*{z) +6e. 

QED 



We can now conclude the proof of theorem I3.4[ 

Proposition 3.16. For every state z in Z and e > 0, there exists a control u in 
U and To such that for every T > Tq, jriz, u) < V*{z) + e. 

Proof : Fix a > 0. 

For every positive integer i, put Si = Define Mj = M{ei) and Ki = K{ei) 
given by lemma [3.151 for £j. Define also Ui = Ma.x{Ki, Mi+i^ > x_ 
We have : Wi > l^Wz e Z,3 m{z, i) < Mi, 3u G U, s.t. 

i^m{z,i),nA^,u) < V*{z) + and V*{y{m{z,i) +ni,u,z)) < V*{z) + —. 

We now fix the initial state z in Z, and for simplicity write v* for V*{z). We 
define a sequence {z\mi,u^)i>i by induction : 

• first put z^ = z, mi = m{z^,l) < Mi, and pick in U such that 
iymi,nii^\u') < V*iz') + §, and V*iyimi + ni,u\z^)) < V*iz') + f. 
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• for i > 2, put = ?/(mj_i + nj_i, \ rrii = 'm{z\i) < M^, and pick u'^ 
in such that < V*{z') + ^ and V*{y{mi+ni,u\ z')) < V*{z') + §. 

Consider finally u'mU defined by concatenation : first is followed for time 
t in [0, mi + rii), then is followed for t in [mi + ni,m2 + ^22), etc... Since 
= y{mi_i + rij-i, M*""*^, for each i, we have + rij, u, z) = z'^ for 

each i. For each i we have nj > Mj+i/a > mj+i/a, so an interval with length rii 
is much longer than an interval with length rrii^i. 

^ length nil length rii length rrii length rii 



For each z > 1, we have < V*{z'-^) + ^. So V*{z') < +^ + ^...+ 



^ + V*{z^) <v* + a-§.^o Um^,nAz\u') < V* + a. 



Let now T be large. 

- First assume that T = mi + ni + ... + mj_i + + r, for some positive i 
and r in [0,mj]. We have : 



1 '•^ 



Iriz^u) = J h{y{s,u,z),u{s))ds 



< 



But mj < arij^i for each j, so 

7t(2,m) < + 2a + — . 

- Assume now that T = mi + ni + ... + mi-i + nj_i + + r, for some positive 
i and r in [0,?T,j]. The previous computation shows that : 

fT-r 

h{y{s, u, z),u{s))ds < mi + (T — r)(t>* + 2a). 



Since z/m,,n,(^\ < v* + a, we obtain : 

T'^t{z,u) = I h{y{s,u, z),u{s))ds + / h{y[s,u, z),u{s))ds, 

Jo JT-r 

< mi + {T -r){v* + 2a)+r{y* + a), 

< mi + T{v* + 2a). 

Consequently, here also we have : 

1t[z,u) <v +2a + —. 
This concludes the proofs of Proposition 13.161 and consequently, of Theorem 

El 
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QED 
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